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C/5 ■ Abstract 



We find the complete branching law for the restriction of comple- 
mentary series representations of 0(1, n + 1) to the symmetric subgroup 
0(l,m + 1) x 0(n — m), < m < n. The decomposition consists of a con- 
tinuous part and a discrete part which is trivial for some parameters. The 
continuous part is given by a direct integral of principal series represen- 
tations whereas the discrete part consists of finitely many complementary 
series representations. The explicit Plancherel formula is computed on 
the Fourier transformed side of the non-compact realization of the com- 
plementary series by using the spectral decomposition of a certain hyper- 
geometric type ordinary differential operator. The main tool connecting 
this differential operator with the representations are second order Bessel 
operators which describe the Lie algebra action in this realization. 
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Introduction 

In the unitary representation theory of reductive Lie groups one is mainly con- 
cerned with the following two problems as was advocated in [5]: 

(1) Classify all irreducible unitary representations of a given reductive Lie group, 

(2) Decompose a given unitary representation into irreducible ones. 

While problem (1) is a long-standing problem in general, there is a classification 
of all irreducible unitary representations for certain subclasses of groups, among 
them semisimple Lie groups of rank one (see [TJ [7] ) . Our focus is on the rank 
one group G = 0(1, n + 1), n G N, for which we study problem (2). 

All irreducible unitary representations of G are obtained as subrepresenta- 
tions of representations induced from a parabolic subgroup P = MAN on the 
level of (g, if) -modules. Up to conjugation P is unique and M = 0(n) x (Z/2Z), 
A = M + and N = M n . We restrict our attention to representations induced from 
characters of P. Denote by TTa,e' n the representation of G, which is induced 
from the character of P given by the character a G C of A and the character 
e G Z/2Z of the second factor of M = 0(n) x (Z/2Z) (normalized parabolic 
induction). In this parameterization 7r^e is irreducible and unitarizable if 
and only if a G iM. U (—71, n). By abuse of notation we denote by ir&} £ 1 ' n+1 * > also 
the corresponding irreducible unitary representations. For a G iM. the repre- 
sentations 7r^e 1 ' n+1 ^ are called unitary principal series representations and for 
a G (—n,0) U (0,n) they are called complementary series representations. We 
have natural isomorphisms ft®^'™ = tiv^ 1 '" for a G iRU (— n, n). 

In this paper we solve problem (2) for the restriction of n°s' n+1 \ a G 
iM. U (— n, n), e G Z/2Z, to any symmetric subgroup of G. By Berger's list P] 
any non-trivial symmetric subgroup of G is either conjugate to 

K = 0(1) x 0(n + 1) or 

H = 0(1, m + 1) X 0(n — m), < m < n. 

Since if is a maximal compact subgroup of G the branching law for the re- 
striction of 7r^ ( 1 ' rl+1 ) i x is simply the A'-type decomposition (jl.2|) which is 
well-known. The branching to H is the main topic of this paper. In the formu- 
lation of the branching law we use the convention [0, a) = for a < 0. 

Theorem (see Theorem I4.T[> . For a G iM. U (— n, n) and e G Z/2Z the repre- 
sentation ir^ e of G = 0(l,n + l) decomposes into irreducible representations of 
H = 0(1, in + 1) x 0(n — m), < m < n, as follows: 

G I ~ V^ | / 0(l,ro+l) - 

jezn[o, |w| -; +m - 2fc ) / 

where H k (M n ~ m ) denotes the irreducible representation of '0(n—m) on the space 
of solid spherical harmonics of degree k on M. n ~ m . 
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The explicit Plancherel formula is given in Theorem 14.11 First of all, the 
restriction ir„ e \h is decomposed with respect to the action of 0(n — m), the 
second factor of H. Then the decomposition of each "H fc (IR™ _m )-isotypic com- 
ponent into irreducible representations of 0(l,m + 1) contains continuous and 
discrete spectrum in general. The continuous part is a direct integral of uni- 
tary principal series representations 7T^V™ +1 \ The discrete part appears if and 
only if k < I Re^|-»+™ an j ^ g a g n ^ c cji rec t sum of complementary series rep- 
resentations. Therefore the whole branching law of ir^ ' £ \h contains only finitely 
many discrete components and the discrete part is non-trivial if and only if 
| Re cr | > n — m. In particular for m > there is always at least one discrete 
component if a is sufficiently close to the first reduction point n or —n. 

For a G iM. the decomposition is purely continuous. In this case the branch- 
ing law is actually equivalent to the Plancherel formula for the Riemannian sym- 
metric space 0(1, m+l)/(0(l) x 0(m+l)) (see Appendix lA"]) and therefore easy 
to derive. We remark that a similar method was used in [§] for the branching 
laws of the most degenerate principal series representations of GX(n, R) with 
respect to symmetric pairs. However, for the complementary series represen- 
tations, i.e. a G (— n, 0) U (0,n), the decomposition cannot be obtained in the 
same way. 

The proof of the Plancherel formula we present works uniformly for a G iRU 
(— n, n). It uses the "Fourier transformed realization" of 7r^ e on L 2 (R™, Ro a dx). 
For this consider first the non-compact realization on the nilradical N of the 
parabolic subgroup P opposite to P. We then take the Euclidean Fourier trans- 
form on 77 = M" to obtain a realization of ir% e on L 2 (R™, \x\~ Re,T dx). The 
advantage of this realization is that the invariant form is simply the L 2 -inner 
product. The Lie algebra action in the Fourier transformed picture is given by 
differential operators up to order two, the crucial operators being the second 
order Bessel operators studied in [SJ \T^\ . Using these operators we reduce the 
branching law to the spectral decomposition of an ordinary differential operator 
of hypergeometric type on L 2 (M + ) (see Section [2]). The spectral decomposi- 
tion of this operator is derived in Section [3] and used in Section 0] to obtain 
the branching law and the explicit Plancherel formula. An interesting formula 
for the intertwining operators realizing the branching law in the non-compact 
picture on N is computed in Section [SJ These intertwining operators will be 
subject of a subsequent paper. 

Up to now only partial results regarding the branching of ir„ £ , cr G (— n, n), 
to H were known: 

• For n = 2 and m = 1 the full decomposition was given by Mukunda |13j 
using the non-compact picture. This case corresponds to the branching 
law 51,(2, C) \ 5X(2,R). 

• For n > 2 and m = n — 1 Spch-Vcnkataramana [HI Theorem 1] proved 

the existence of the discrete component 7r^i\' re ^ in 7iv ™ for a G (1, n) 
(special case j = k = in our Theorem). They also use the Fourier trans- 
formed picture for their proof. This is a special case of their more general 
result for complementary series representations of G on differential forms, 
i.e. induced from more general (possibly non-scalar) P-representations. 
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• The same special case was obtained by Zhang P33 Theorem 3.6]. He actu- 
ally proved that for all rank one groups G = SU (1, n + 1; F), F = R, C, H, 
resp. G = ^4(_2o) certain complementary series representations of H = 
SU(l,n;¥) resp. H = Spin(8, 1) occur discretely in some spherical com- 
plementary series representations of G. His proof uses the compact picture 
and explicit estimates for the restriction of K -Quite vectors. 

Acknowledgements. We thank Toshiyuki Kobayashi and Bent 0rsted for 
helpful discussions. Most of this work was done during the second author's visit 
to Aarhus University supported by the Department of Mathematics. 

Notation. N = {1,2,3,. . .}, N = NU {0}, R+ = {x G R : x > 0}. 

1 L 2 -realization of the complementary series of 

0(1,71+1) 

In this section we recall the necessary geometry of the group G = 0(1, n + 1) 
and some of its representation theory. 

1.1 Subgroups and decompositions 

Let G = 0(1, n + 1), n > 1, realized as the subgroup of GL(n + 2,R) leaving 
the quadratic form 

R n+2 — > R, x=(xi,..., x n+2 y h> x\ - (x\ + ■ ■ ■ + x 2 n+2 ), 

invariant. We fix the Cartan involution of G given by 6(g) = g~* = (<?') _1 , 
g € G, which corresponds to the maximal compact subgroup K := G e = 0(1) x 
0(n + 1). On the Lie algebra level the Lie algebra g of G has the Cartan 
decomposition g = t © p into the ±1 eigenspaces t and p of 8 where £ is the Lie 
algebra of K . Choose the maximal abelian subalgebra a := M.H C p spanned 
by the element 

H := 2(E\ >n+ 2 + E n+ 2.i), 

where Eij denotes the (n + 2) x (n + 2) matrix with 1 in the (i,j)-entry and 
elsewhere. The root system of the pair (g, a) consists only of the roots ±27 
where 7 £ a£ is defined by j(H) := 1. Put 

n:=02 7 , n:=g_2 7 = #ri 

and let 

7V:=cxp G (n), N := exp G (n) = ON 

be the corresponding analytic subgroups of G. Since dim(n) = dim(n) = n the 
half sum of all positive roots is given by p = nj. We introduce the following 
coordinates on N and N: For 1 < j < n let 

Nj := Eij + i + — Ej + i, n+ 2 + E n+ 2,j+i, 

Nj := Eij + i + + Ej + i t7l+ 2 — E n+ 2,j+i- 
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For let 

n n 

n x := cxp ( ^ x jNj^J £ iV, := exp ( ^ XjN^j 6 TV. 

j'=i i=i 

Further put M := Zk{o) and A := exp (a) and denote by m the Lie algebra of 
M. We write M = M+ U m M + where 

M+ := {diag(l, k,l):k€ O(n)} = O(n) and 
mo := diag(-l, 1, . . . , 1, -1). 

Via conjugation the element mo acts on N and N by 

ffio^ifo" 1 = n -i and mon x m,Q = ri— x 

and the action of m £ = 0(n) on TV and N by conjugation is given by 

mn^m -1 = n ma: and mn x m = n mx 

for x £ M n , where mi is the usual action of 0(n) on R™. Further A acts on N 
and TV by 

e tH n x e~ tH = n e 2t x and e tH n x e~ tH = n e -it x 

for a; € R n , f£R. The following decomposition holds 

= n®m0a®n (Gelfand-Naimark decomposition). 

The groups 

P := MAN and P := MAN = 6{P) 

are opposite parabolic subgroups in G and NP C G is an open dense subset. Let 
W := Nk{o)/Zk{o) be the Weyl group corresponding to a. Then W = {1, [wq]} 
where the non-trivial element is represented by the matrix 

w = diag(-l, 1, . . . , 1) e K. 

The element wq has the property that wqNWq = N and hence wqPwq = P. 
More precisely, 

wqtIxWq 1 ~n- x and WQe tH WQ 1 — e~ tH . 

We have the disjoint union 

G = P U PwqP (Bruhat decomposition). 

The following lemma is a straightforward calculation: 

Lemma 1.1. For x € W 1 , x ^ 0, we have Wq n x = n y me tH n z € NP with 
V = ~\x\~ 2 x, / _x 



\x\~ 2 x, m= I l n -2\x\~ 2 xx t 



t = \og\x\, 



-1 
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Let r be the involution of G given by conjugation with the matrix 

diag(l m , -ln-m, !)• 

Then the symmetric subgroup H := G r is isomorphic to 0(1, m + 1) x 0(n—m). 
The subgroup H is generated by the subgroups Nh, Njj, Mh and A, where 
(viewing R m as the subspacc R m x {0} C f) 

Afc := {n x : x G M m } and 7V H := {n* : x G JR m } 

and Afjj := U m M^ with 

M^j := {diag(l, fci, k 2 , 1) : fci G O(m), fc 2 G 0(n - m)} = O(m) x 0(n - m). 

Also denote by 

P H ■= M H AN H and P H := MrANh 

the corresponding parabolic subgroups. We write f) for the Lie algebra of H . 

1.2 Principal series representations — non-compact pic- 
ture and standard intertwining operators 

We identify with C by A i— > A(P), i.e. a G C corresponds to 177 € a£. Under 
this identification p corresponds to n. For <r G C let e CT be the character of A 
given by e a {e tH ) = e' 7 *, tGl. Further, for e G Z/2Z denote by £ £ the character 
of M = M + U m M + with ^(to ) = (-1) £ and £ e (m) = 1 for m G M+. For 
cr G C and e G Z/2Z we consider the character x<r,e := £,e ® e CT ® 1 on P = MAN 
and induce it to a representation of G: 

7£ E := Ind£( X(r , £ ) 

= {/ G C°°(G) : f(gman) = f e (m) _1 o _,T - p '/(flr)V 5 G G,man eP = MAN}. 

The group G acts on P/ e by left-translations and this action will be denoted by 
7r^ e . Since NP C G is dense, a function in is already uniquely determined 
by its values on N and for / G I„ e we put 

f w (x) := /(rU x G M". 

Let J^ e := {/y : / G P/ e } and denote by tt^ e the corresponding induced action, 
i.e. 

In view of the Bruhat decomposition G = P U PwqP this action can be com- 
pletely described by the action of P and wq. Using Lemma ll.ll we find 

n% s (n a )f(x)= f(x-a), n a eN, 

TT^Jm)f(x) = f{mr x x), m G M+ = O(n), 
< £ (m )/(x) = (-l) £ /(-x), 

< e (e'")/0r) = e^>V(e 2 H e tfl e4, 

< e K)/(x) = (-iy\x\-°- n f(-\x\- 2 x)- 
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This also gives the following expressions for the differential action dn^ = dn^ e 
of the Lie algebra g, which is independent of e: 

d^(N j )f(x) = --^-(x), j = l,...,n, 

dwf(T)f(x) = -D Tx f(x), Tem^so(n), 
d*°(H)f(x) = (2E + a + n)f(x), 



(N j )f(x) = -\x\ 2 ^-(x 



where D a denotes the directional derivative in direction a € K" and E = 
x j~Bx~ * s the Eulcr operator on R™. For the action of n we have used 

the identity dvif (i\Q = ^Jw^dn^ (N^ a )7r^ e (w^ J ). 

Now suppose a G (0, n) and consider the Knapp-Stcin intertwining operator 

J(<r,e) ■ I°e "> T-a,e g iven by 

J(a,e)f(g) := _f{gw n)dn, geG, f £ I° e , 

J N 

where dn is the Haar measure on N given by the push-forward of the Lebesgue 
measure on l n by the map R™ — > N, x n x . This intertwining opera- 
tor induces an intertwining operator J(a,e) : — > I_ a£ by J(er, e)fjj ■= 
(J(ct, e)/)]y, / <G I^.e- Using Lemma 1X7X1 we obtain 

J(v,£)fw( x ) = / f(n x w n z )dz 

= (-lf [ \z\-°- n fjf{x-\z\- 2 z)dz. 

Consider the coordinate change y := x — \z\~ 2 z. Its Jacobian |det(|^)| is 
homogeneous of degree — 2n, 0(n)-invariant and has value 1 for z = e\. Hence 
it is equal to |z|~ 2 ™. This finally gives 



J(v, e)f(x) = (-iy J \x ~ yr n f(y) dy 

= (-i) s (i-r n */)(x), 

so J{a,e) is up to sign given by convolution with the function | — |°" _ " . We 
define a G-invariant Hermitian form (— |— ) CT)£ on by 



(f\g)..e:=(-inf\J(<J,e)g) L 2 iMn) = / / \x - y\°- n f{x)g{y) dx dy. (1.1) 

For a € (0, n) this form is in fact positive definite and in this case the closure 
W^e 01 foe w ^ tn res P ec t to the inner product (— |— ) CTj£ gives an irreducible uni- 
tary representation (H^ e ,Tr^ e ) of G. Using the intertwining operator J(cr,e) : 
I® e — > I^ as one also obtains a unitarization (JiP_ a e , 7r5 CT e ) — (H^ 6 ,7r^ e ) of 
I 6 . 

— <T,£ 

For a E iW. the usual L 2 -inner product provides unitarizations (H^ s , ^ e ) on 
e = L 2 (M. n ) and these representations form the unitary principal series. For 
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a € (— n, 0) U (0,n) they comprise the complementary series for G. Note that 
for any a <E (— n,n) U the analytic continuation of the operator J(a,e) pro- 
vides an intertwining operator between the irreducible unitary representations 
(^ £ ,< e )and (^,7^). 

From the compact picture it is easy to see that the K-type decomposition 
of the representations iT a ^ is given by 



*%\k 



Y^n k {w n+1 ), (1.2) 



k=0 



where 0{n + 1) acts as usual on H k (W l+1 ) and O(l) acts by (— l) e+k , giving 
combined the action of K = O(l) x 0(n + 1). 

1.3 The Fourier transformed picture 

Consider the Euclidean Fourier transform F-gn : S'(M. n ) — > <S'(R") given by 

= (2tt)-9 / e-^^w^dy. (1.3) 



For <t G (— n, n) U iR and e € Z/2Z we define a representation of G on 

J-R-Xe by 

C(s)oJ r =^o4( s ), 9 eG. 
It is easy to calculate the group action of P = MAN: 

Pa.e(n a )f(x) = e^W/Or), n a € JV, (1.4) 

Pa, e (m)/(a:) = /(m^x), m G M+ = 0(n), (1.5) 

p CT , E (m )/(z) = (-l) £ /(-z), (1.6) 

p CT)£ ( e tir )/(x) = e^-^VCe" 2 ^), * G K. (1.7) 

The action of wq in the Fourier transformed picture is more involved (see e.g. 
(TT1 Proposition 2.3]). Note that by these formulas the restriction p a ,e\-p also 
acts on C°°(R m \ {0}). Using the classical intertwining relations 



Xj o J" R ™ = J" Rn o (-ig^r), 
g|r o J"k« = J" R n o (-ixj) 

it is easy to compute the differential action dp^ of p^ £ 



dp°(N s )f{x)=ixjf(x), j = l,...,n, (1.8) 

dp°(T)f(x) = -D Tx f{x), Tem-so(n), (1.9) 

dp°(H)f(x) =-(2E-a + n) f(x), (1.10) 

dp°{Nj)f{x) = -iB?" f{x), j = 1, . . . ,n, (1.11) 

where we abbreviate 

*T :=a; J A-(2^- ( r + n)A 
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The operators 23™' CT are called Bessel operators and were for G = 0(1, n + 1) 
first studied in [$] • They are polynomial differential operators on R n and hence 
the action dp^ defines a representation of g on C°°(f2) for every open subset 

n c m™. 

To describe the representation spaces T^'H^ e in the Fourier transformed 
picture we recall that the Fourier transform J-'s.n intertwines convolution and 
multiplication operators. Further, the Riesz distributions R\ £ <S'(R n ) given by 

(R x ,<p)=*L-[ <p(x)\x\ x 6x, ip£S(R n ), 

1 (— — ) JR™ 

for Re A 3> and extended analytically to A £ C satisfy the following classical 
functional equation (see [4[ equation (2') in II. 3. 3]) 

FriRx = R-\- n . 

With this and we see that in the Fourier transformed picture the represen- 
tations p% e are realized on T^U% e = L 2 (R n , \x\~ RoCT dx), a £ iR U (-n,n), 
and the Fourier transform is a unitary (up to scalar multiples) isomorphism 
•7-r*» : L 2 (R", Rccr da;) — > e intertwining the representations p^ £ and 
7r^ e . The standard intertwining operators J(a,e) are in this picture given by 
multiplication 

L 2 (R n ,\x\- Rc,T dx) -> L 2 (R",|a;| Rc,T dx), /(x) h> |s| _< 7(a:). 

The if- type decomposition (|1.2p is difficult to see in the Fourier transformed 
picture. However, one can still explicitly describe the space of infinite vectors. 
For this recall the renormalized if -Bessel function K a (z) from Appendix IB. 1 1 
It is easy to see that the vector 

tf(x):=K- f (\x\), xeR n \{0}, (1.12) 

is t-fixed and constitutes the minimal 6-type. Note that as if-representation the 
minimal if -type is for e =/= not the trivial representation since mo £ if acts by 
(— 1) £ . To describe the underlying (g, if )-module we denote for / £ C°°(R + ) 
and k £ N by / ® |a;| 2fc the function f(\x\)\x\ 2k and by / <g> |a;| 2fc C[a;i, ...,x n ] 
the space of all functions of the form f(\x\)\x\ 2k p(x) for some polynomial p £ 
C[xi, . . .,x n ]. 

Lemma 1.2. The underlying (g, K)-module of (p a ,e, L 2 (R n , \x\~ Rocr dec)) is 
given by 

oo 

L 2 (R n , \x\- Re °dx) K = K_ f+k ® lafCfri, . . . ,ar„]. (1.13) 

fc=0 

Proof. Since g = t + a + n the universal enveloping algebra W(g) of g decomposes 
by the Poincare-Birkhoff-Witt Theorem into U{q) = U{n)U{a)U{t). The (g, K)- 
modulc L 2 (R™, \x\~ Ro<T dx)n is generated by the t-fixed vector ^ and hence 

L 2 (R n ,\x\- Rc °dx) K =U(q)^ =W(n)W(o)V°. 



9 



By dHJ and (pTUj) we have U(n) = C[x u ...,x n ] and U(a) = C[E}. Using 
(|B.3p we further find that the Euler operator E acts on functions of the form 
K a (\x\)\x\ 2k , a G R, fc G N 0l by 

£ (iMM)M 2fc ) = -i^ Q+1 (|x|)|x| 2fc+2 + 2fc^ Q (|x|)|x| 2fe 



Hence 



U(xC]U(a)^ = U(n) C (£_ 5+fc ® M 2fc ) 



OO 



0iT_ f+fe (8)| ; c| 2fe C[x 1) ...,a; T[ ] 



fe=0 

and the claim follows. □ 



2 Reduction to an ordinary differential operator 

This section deals with the reduction of the branching problem for p^ J\h to an 
ordinary differential equation on R + . 

Consider the L 2 -realization L 2 (R™, \ (x, y)\~ Rc<T dxdy) of the representation 
p^ e where we split variables (x,y) G R™ x R Il_m . We realize unitary prin- 
cipal series and complementary series representations p^ g' m+ of the first 
factor 0(l,m + 1) of H = 0(1, m + 1) x 0(n — m) in the same way on 
L 2 (R m ,|a;|- Rer da;). For the second factor 0(n - m) denote by H k (R n - m ) 
its representation on solid spherical harmonics on R™~ m of degree fc 6 No by 
left-translation. 

Proposition 2.1. Let a G (—71,11) U iK. and r E (— m, m) U iR. For every 
solution F G C°°(R+) of the second-order ordinary differential equation 

t(l + t)u"(t) + ( -°+2k+n- m +2 t + 2fc+n-m ) ^ 

+ i(( -^ + "- m ) 2 -(i)>W = o 

which is regular at t = £/ie map 

* : C°°(R m \ {0}) ®H k (R n - m ) -> C°°(R n \ {.t = 0}), 

*(/® y) := M — - 2 2 fc - +m F(]sf)/(x)0(y), 

is Ph- and \)-equivariant i/C oo (R" l \{0}) carries the representation p^l^™" 1 " 1 ^ |p 
(resp. dpr m+ ) and C°°(R" \ {x = 0}) £/ie representation p^ s \-p (resp. 

Proof. Put p := 2k + n — m and a := — p 1 ^ so that 

Since f) = rijy + m^r + a + n# it suffices to check the intertwining property for 
N H , M H , A and n H - 



10 



(i) For n a G N H both p° E {n a ) and p°£$ +l) (n„) are by (HI the multipli- 
cation operators e 1 ^'") and hence the intertwining property is clear. 

(ii) Let to = diag(l, ki, k 2 , 1) € M^, fci e 0{m), k 2 € 0(n — m). Then with 
to' = diag(l, fci, l„_ m+ i) we have by (|1.5[) 

^ e (m)f (/ <g> y) - *(/ <g> ^(fcf 1 *. *fe ^ 

Further, for mo we have with (|1.6[) 

4(mo)*(/ ® M*. tf) = ® -») 

= (-iy\(- x )\°F($E$)f(-x)<t>(-y) 

= (-iy+*\ x \*F($)f{-x)<t>(y) 

(iii) For a = e tH £ A we obtain with (fTTT)) 

p£ e (o)*(/ ® y) = e<*- n >**(/ ® ^(e" 2 ^, e" 2 ^) 

= e(— ) t | e - 2t xr J F(^4)/(e- 2t x)0(e- 2t 2/ ) 
= e( CT -"- 2Q - 2fc )' F(^)/(e- 2 *x)0( 2/ ) 
= e^-^xFF^Me-nxMy) 

(iv) To show the intertwining property for xih it suffices by (jl.lip to show the 
identity 

for j = 1, . . . , m which follows from the next lemma. □ 
For a, \x € C we introduce the ordinary differential operator 

„ . , d 2 /ri — a + 2 u\ d . 

Lemma 2.2. Leta,r,a£ C. fc € N , F G C°°([0,oo)), / € C°°(R m \{0}) and 
4> G % fe (R n— m ). T/ien for every j = 1, . . . ,m we have 



j 



\x\ a F(^)f(x)ct>(y)\ = \x\ a F(^)B™»Tf(x)<l>(y) 

+ xAx\ a - 2 f{x)4>{y) (4V a ,, +a(a-n- a)) F(^). 
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Proof. We first note the following basic identities, where J| and are the 
gradients in x £ K m and y £ M. n ~ m respectively, and A x and A y the Laplacians 
on R m and R Tl_m respectively: 



d_ 

dx 



\x\ a = a\x\ a - 2 x, 



A x \x\ a = a(a + m - 2)\s 



JLp(\y\l) - _M F '(lvil) x A F(^) - A^- F"(^) - 2(m - A)^F'(^) 

9j/ l 1 ^ - b| 2 ^l 1 ^ ^y r \\x\z)- | x |4 ^ M^l 2j+ |a;|2 r \\x\*)- 

The calculation is split into several parts. In what follows we abbreviate t := 

\y\ 2 

W 

(i) We begin with calculating XjA x ^>(f (g) <j>): 

XjA x ^(f^(p)(x,y) 
= *( Xj A x f ® 0)(x, y) + ^A^l^r • F($)f(x)4>(y) 

+ Xj A x F(^) ■ \x\ a f(x)<Kv) + 2x i^~ ■ • 

+ ■ ~sF ■ + 2 *T ■ i (l > ■ 

= ty(xjA x f <g> y) + ^^["-^/(a:)^^) (-4iF'(t) + 2aF(t)) 

+ ^|x| Q - 2 /(a;)^(y) (4t 2 F"(t) - 2(2a + m - 4)tF'(t) +a(a + m- 2)F(t)) . 

(ii) Next we calculate XjA y ^(f ® </>): 

XjA y y(f®(j))(x,y) 
= Xj A y F(^) ■ \x\ a f(x)4>{y) + xjAy^y) ■ \x\«F{$)f(x) 
Ml 



dy dy 



2x j Z 'IT.- M°7(s) 



Xj\x\ a - 2 f(x)(j)(y) (4tF"(i) + 2(2k + n- m)F'{t)) 



since E<j> = k<j> and A y <f> = 0. 

d 

dxj 



(hi) We now calculate -Sr^iJ 



d 

—y{f®<t>){x,y) 

■ F($)fix)<Kv) + —gj- ■ \xn(x)4>(v) 

+ j£(x)-\x\"F($)4>(y) 

%L(x) ■ \x\ a F(^(y) + xM^m^y) (-2tF'(t) + aF(t)) 
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(iv) Next we find (2E - a + n) g | T *(/ <g> <j>) by using (iii): 

(2E-a + n)-£-*(f®<j>)(x,y) 
dxj 

= (2E-a + n + 2(q + k))^L{x) ■ \ x \<*F(^)cj>{ y ) 

+ 2x ] \x\ a - 2 Ef{x)4>{y) (-2tF'(t) + aF(t)) 

+ (2(q + k-l)-a + n)x j \x\ a - i f{x)^>{y) (-2tF'{t) + aF(t)) 

since E\x\P = P\x\ p , EF(^Z) = and E<j) = k<j). 
Now, putting (i), (ii) and (iv) together gives the claimed identity. □ 

3 Spectral decomposition of a self-adjoint second- 
order differential operator on 1R + 

In this section we find the spectral decomposition of the second-order differential 
operator T> a ^ on L 2 (R + ,t L ^~ (1 + t)^ 3 ^ dt) using the theory developed by 
Weyl-Titchmarsh-Kodaira (see [TTlfTo] !. 

We fix a e iR U (0,n) and e € Z/2Z. (In the case a G (-n,0) only the 
derivation of the discrete spectrum in Section [23] is slightly different. However, 
since 7r CTlE = K-a. £ the decomposition of the representations is again the same and 
it suffices to consider a £ iRU(0, n).) Further fix fc € No and put fj, := 2fc+n— m. 
We assume that m < n so that /i > 0. Proposition 12.11 suggests that the 
decomposition of the 0(n — m)-isotypic component of T-L k (M. n ~ m ) in p^ £ into 
irreducible 0(1, m + ^-representations is given by the spectral decomposition 
of the differential operator V a ^ defined in (|2.1|) . Writing 



V^ = t(l+t)^ + ((a + b+l)t + c)^ 
with 



a — /i T (7 — /! T /X 

a = 1 — , o = , c = — , 

4 4' 4 4' 2' 

it is easy to see from (|B.4[) that the hypergeometric function 

F{t, t) := 2 F t (a, b; c; -t) (3.1) 

solves the equation 

P CTl(tl M + \*u = 0, A*=a6=^ g 4 ^ ^ _ (i 

We find a spectral decomposition of T> a ^ in terms of F(t, r). 
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3.1 Simplifications 

Following [161 Examples 4.17 & 4.18] we first make the transformation t 
sinh 2 (|). Using t4: = tanh(§ )-n- we write the operator T> a ^ as 

d 2 _ d 



with 



Putting 

y(x) = r(.T) _1 u ^sinh 2 (j^jj with r(a;) = sinh (^—^j cosh ^— 
we finally sec that the differential equation T> a „u + X*u = is equivalent to 



dfy 

da; 



^ + (X* -q*(x))y = 



2 



with 

= O*- 1 ^- 3 ) tanh f - ^~ 2 ) + 1 + + tanh (f 

16 \2/ 8 16 V2 

To stay in line with [16j Examples 4.17 k, 4.18] we shift the eigenvalues by 
putting q(x) := - (^) and A := A* - (^) and obtain 

g + (A-g(s))i, = 0. (3.2) 

Note that q(x) is real-valued for <j € iR U R and hence the operator -ri — 
is formally self-adjoint on L 2 (R + ). 



3.2 Singularities and the boundary condition 

The differential equation (|3.2[) has regular singular points at x = and x = oo. 
The corresponding asymptotic behaviour of solutions at x = is given by x 1 ^ 
and a;"^^ - for /i 7^ 2 and by j5 and log(x)a;2 for fi = 2. Hence x = is of limit 
point type (LPT) if fi > 4 and of limit circle type (LCT) if fi = 1,2,3. The 
solution 

771(2;, A) = r(x)~ 1 2 Fi(a,b;c;-smh 2 (%)) 

ii-i 

has asymptotic behaviour x~ ~ near x — 0, where 

a — 11 . rr , a — U . /- u 

a= - 1-ivA, 0= : IV A, c = — . 

4 4 2 
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Note that 771(2:, A) is holomorphic in A <G C and real- valued if A £ R and tr £ 
R U «R. Indeed we can easily see i]i(x, A) = 771(2;, A) for A 6 1 and <r 6 K U iK 
using Rummer's transformation formula (|B.7[) . In the case of (LCT) at x = 
we impose an additional boundary condition (which is automatic in the case of 
(LPT)). For this we use the solution 771(2;, A) for a fixed Ao and impose 

Um%(-A ),ii)(a;) = 0, (BC) 

x— >0 

where W(u, v) = u'v — uv' denotes the Wronskian. Then in both (LPT) and 
(LCT) cases r]i(x, A) is the unique solution of (|3.2[) which is L 2 near x = and 
satisfies the boundary condition (|BC[) . Near x = oo we consider the solution 

m (x, A) = r(x)- 1 sinh- 26 (|) 2 F 1 (&, 6 - c + 1; 6 - a + 1; - sinlT 2 (§ )), 

which has the asymptotic behaviour e lx ^ near x = oo and hence is L 2 near 
x = oo for < arg(-\/A) < 7r. The other solution is obtained by interchanging a 
and b and has asymptotics e -' lx ^ whence x = oo is always of (LPT). Altogether 
the operator in (|3.2[) extends to a self- adjoint operator on L 2 (R + ) under the 
boundary condition (|BCI) . 

3.3 Titchmarsh— Kodaira's spectral theorem 

We calculate the Wronskian 



r{x)- 2 W{ 2 F l {a, b; c; - sinri (f )), simi (^) 2j Fi(&, b- c+l;b- a + 1;- smli (^))(a;) 
r{x)- 2 sinh(f ) cosh(f )W( 2 F 1 (a, b; c; -z), z'^F^b, b - c + 1; b - a + 1; -I))(sinh 2 (§ )) 



r(x) 2 sinh(f) cosh(|) 



T(b- q)r(c) 

r(6)r(c- a ) 



x W(z- a 2 -Fi(a, a-c+l;a-6 + l; z-\F 1 {b, b - c + 1; 6 - a + 1; -I))(sinh 2 (§ )) 



(a -6) 



r(6-a)r(c) 



2i7Ar(-2iv / A)r(f) 



r(6)r( c -a) r(-^ -n/X)r(£±H -Va)' 

which depends only on A. Now for / £ L 2 (R + ) real- valued define 
1 



<f>(x,X) 



W{ m , m ){X) 
Then by [TBI Section 3.1] we have 

(*oc-\-i5 



772(2;, A) / 77i(y,A)/(7/)d7/ + ?7i(a;,A) / 772(7/, A)/(y) dy 



fix) = lim / $ (x, A) dA = lim / Im<& (x, A) dA 



3.4 The continuous spectrum 

We first treat the integration over the interval (0,oo). In this case Vx £ R+. 
We study the cases a £ (0, ra) and a £ iR separately. Recall that in both cases 
771 (x, A) 6 R. 
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(i) er <= (0, n). In this case a = b and c € K. Using T(z) = r(^r) we obtain 



Im 



T{b)T{c-a) 



2ir(x) 



(a-b)T(b-a)T(c) 



sinh- 2b (f ) 2 F 1 (b, b-c+l-b-a+1-- sinlT 2 (§)) 



sinh- 2a (f ) 2 Fi(a, o-c+1; o-6+l;- sinrr 2 (§ )) 



(6 - o)r(o - 6)r( c ) V2 

1 T(a)T(b)T(c-a)T(c-b) 
2i{a-b)r(x) T(a - b)T(b - a)T(c) 2 



2j F 1 ( a ,6;c;-sinh 2 (f)) 



1 



2i(a - b) 
1 



r(a)r(c - 6) 



771 (z, A) 



r(a - 6)r(c) 
r(-^ + iVA)r(^ + n/A) 



4VX 
whence 

Im$(ir, A) = - 



r(2i>/X)r(f) 



771 (a;, A) 



4\/A 



r(-^ + iVX)r(^ + 4 VA) 



r(2iv / A)r(f) 



?7i(.t,A) / 771 (y, X)f(y)dy. 
(3.3) 



(ii) cr 6 iM.. Note that a = c — a, b =- c — b and c <E R. Hence we find by 
Rummer's transformation formula (IB. 71) 



Im 



/ 7/ 2 (a;, A) 
^(t?!, 772) (A) , 

r(6)r(c-a) 



2ir(x) [(a - b)T(b - a)T(c) 
T(c-b)T(a) 



sinh- 2b (f ) 2 F 1 {b, b-c+l;b-a + l;- sinh- 2 (f )) 



COShCT (2) (6 _ a)r(a _ 5)r(c) 



sinh" 2(c " b) (f ) 2 Fi(c - b, 1 - 6; a - & + 1; - sinh" 



2ir(x) 



r(6)r(c a) . sinh -2b(|) 2i?i ( 6)6 _ c + 1 . 6 _ a + 1 ._ sinh -2(|^ 



(a - 6)r(6 - o)r(c) 



r(c 6)r(a) . sinh -2a(^ 2Fi ( aa _ c + 1 . a _ 6 + 1 ._ sillh -2^« ) « ) 



(6 - a)T(a - b)T(c) 



771(0;, A) 



where wc have used the same calculation as above. This shows that 
also holds in the case a G iffi. 

In both cases we collect the continuous spectrum 

1 f°° 

- - / Im$(ar,A) dA 

T JO 



47T 



771 (x, A) / Tfr(y,\)f(y)dy) 



T(2iVA)r(f) 



dA 
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3.5 The discrete spectrum 

Next we consider the integration over the interval (— oo,0). Here y/\ G iR+. 
Again we treat the cases a G (0, n) and a G iM. separately. Recall that in both 
cases i]i(x, A) G R for all x € R+. 

(i) cr G (0,n). We have a, 6,c € R which gives 772(0;, A) € R for all x G R_|_. 
The poles of the function $(x, A) as a function of A are the zeros of the 
Wronskian. Since Re( £ ± i£ - iV\) > f > the poles of $(x, A) are all 
simple and exactly at the points where —^-^ — iV\ G —No- This gives 

iV\ = + j and A = - (^-j) 2 for j G N with j < 

Consequently we have b = —j and therefore, by (|B.5|) 

r( a - b)r(c) 
^ l(x ' A) ^i>jf(^) ??2(x ' A) 



r(-£^ + 2j )r(f) 



772 (x, A). 



r(-^+j)r(j + f) 

Using res z= _„r(z) = ( -~^,- > we find 

1 

^-(^-'OVfoi.TfcXA) 

r(f-i) p / <r-/i 

res, ^„_ u A 2l ; 7V A 



( 2 j-^)r(^-2i)r(f) *=-(^ 

r(f-j) ( <?-^\{-iy 

(2j-^)r(^-2j)r(f) V J 2 ) j\ 

(-i) J 'r(§-j) 



i!r(^-2i)r(f; 



Since Im $(x, A) = for A G (—00, 0) not a pole we obtain by the residue 
theorem the discrete spectrum 



1 f iS 

— lim / Im $ (x, A) dA 

reS A=-(^ii- J ) 2$ ( :r ' A ) 



j'e[o 

(-i)'T(f -i)r(-^+j)r(i + f) 



, e[0) ^)- ^ !r (t) 2 W - 2j)r(-^ + 2.7) 



/ m(v,- -i) )/(y)dy. 

Jo 



(ii) a G iR. We have a = c — 6, 6 = c — a and c G R. By Rummer's 
transformation formula (IB. 71) 



772(0;, A) = r(x)- 1 cosh ff (f ) sinlT 2(c " a) (f ) 2 F 1 (c - a, 1 - a; b - a + 1; - sinh~ 
= r(x)" 1 sinh" 2fo (f ) 2 Fi(6 - c + 1, b; b - a + 1; - sinh" 2 (f )) 
= 7/2(2:, A). 
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Further since T(z) = T(z) we also find that the Wronskian W(r)i,r)2) is 
real-valued. Hence Im<fr(x, A) = for A e (— oo,0). Since W (771, 772) has 
no poles in A e (— 00, 0) for a € iM. there is no discrete spectrum in this 
case. 



3.6 The spectral theorem for T> ail 

Together this gives the spectral decomposition of L 2 (R+) into the eigenfunctions 
r]i(x, A). For the precise statement let 



5(a,/i):=(0,oo)U |J 

je[o,s±^=Ji)nz 



a — n 



Note that S(o~, fi) = (0, 00) for a € iK. On S(o~, /j) we define a measure div^ by 



S(o-,/i) 



<7(A)d^(A):=— / 5 (A) 



r(-^ + iVA)r(^ + zVa) 



r(2z V / A)r(f) 



dA 

7x 



E 



(-i)ir(f-j)r(-^ + i)r(i + |) _ 2 

■ \ 9\ \ 4 J- 



j!r(^) 2 r(^ - 2j)r(-^ + 2j) 

Then by [THl Sections 3.1 & 3.7] we have: 

Theorem 3.1. For a £ iRU (0, n) and /x € N the map 



L 2 (M+) -+L'{S(a,ii), di^), f^g(\)= m (x, \)f(x) dx, 

Jo 



is a unitary isomorphism with inverse 



L 2 (S(a,[i), du^)^L 2 (R+), g^f(x) 



r)i(x,X)g(X) dv atfl (X). 



For our application we need the spectral decomposition of the operator T> a ^ 
which follows immediately from Theorem 13.11 by the transformation u(t) t—¥ 
r(x) _1 u(sinh (f )). To state this put 

Tfo/j) :=«+U |J {a - M - 4j} 
je[o, Re 4~" )nz 

and define a measure dm a ^ on T(cx, /j) by 



L 



T(<7, M ) 



jMdm^^r) := 



8?ri 



9(r) 



mm) 



E 



(-i)T(f-j)r(-^+j)r( J + f) 
j!r (f) 2 r(^- 2 j)r(-2=H + 2j) 



dr 



fl (a- M -4j). (3.4) 
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Corollary 3.2. For a e £R U (0, n) and \i G N the map 

L 2 (M + ,^(l + i)- £ ^dt) ^L 2 (T(a,/j), dm^), 

/>oo 

f^g(r)= F(t,T)f(t)t' ! * 1 (l+t)-*dt 
Jo 

is a unitary isomorphism with inverse 

L 2 (T(a ;A 0, dm a ^) ^ L 2 (R + ,t^ (1 + t)- 2 ^ dt), 

S ->/(*) = [ F[t,T)g(T)dm^(T). 

Remark 3.3. For the discrete part, namely for r = a — fi — Aj, j £ No, the 
Gaufi hypergeometric function F(t, r) degenerates to a polynomial in t of degree 
j. More precisely, we have (see (|B.8|I ) 

F(t, a -fi-Aj) = -£-P^-*\l + 2t), 

{ 2 )n 

where Pn" 1 ' (z) denote the Jacobi polynomials. 

Remark 3.4. For a £ (0,n) the results of Corollary 13.21 can also be found in 
[3J formula (A. 11)] where the hypergeometric transform appears (essentially) as 
the radial part of the spherical Fourier transform on SU (1, n) / SU (n) . Since 
our approach provides a unified treatment of both complementary series and 
principal series, including the case a £ iR, we gave a detailed proof in this 
section for convenience. 

4 Decomposition of representations and the Plancherel 
formula 

Using the spectral decomposition of T> a ^ obtained in Corollary 13.21 we find in 
this section the explicit Plancherel formula for the decomposition of p^ £ \h- 

Let us first consider the action of 0(n — m) on L 2 (R n , \(x,y)\~ Rca dxdy) 
which gives the following decomposition as 0(n — m)-representations: 

L 2 (m n ,\(x,y)\-**°dxdy) 

OO 

= L 2 {R m x R +7 {\x\ 2 +r 2 )- B ^r 2k+n - m - 1 dxdr)®H k {R n - rn ), (4.1) 

fc=0 

where r = \y\. We fix a summand for some k € No and put again fi = 2k + n — m. 

2 

The coordinate change t := j^p gives 

L 2 (R m x M + ,(|a;| 2 +r 2 )- S ^r^ 1 d.Tdr) 

= L 2 (R m xR+,i|xr Rcff+ ^^(l + f)-^d.Tdt). 
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Since 

L 2 (R m x M + ,i|xr Rcff+ ' i t ii ^(l + t)- a ^da;dt) 

S L 2 (M"\ ||a?|- Bfl ' + '* da;)§L 2 (R + ,t li ^(l +t)-^ dt) 

we can apply Theorem 13. II to find that the map 
L 2 (R m x M + ,i|xr Rc ' T +' i t li ^(l + t)- a ^dxdi) 



^ L 2 (R ro ,||x|- ReT dx)dm (TlM (T) 

JT(<t.u) 



'T(<x, M ) 

given by 

/>oo 

/(x, t) ^ />, r) := 1x1-^=5^ / F(t, r)f(x, t)t^ (1 + t)"i di 



is a unitary isomorphism, where r) is defined by (|3.1[) and the measure 
dm,^ is given by (|3.4p . Its inverse is given by 



g{x,r) ^ g{x,t) := I \x\ = F(i, r)g(x, r) dm CT) ^(r). 

Now we put these things together. For cr € zK U (0,n) and k £ No we put 
/i := 2/c + n — m and define an operator 

9{tr,k) : I L 2 {W m ,±\x\- RcT dx)dm r7fi (T) \ MH k (R n - m ) 

-^L 2 (R n ,\(x,y)\- Re "dxdy) 

by 

*(a,k) U®4>) (x,y) 

:=|x| £= 5^0(v) / 2 F 1 (^f±^,^f^;f;-^)/(^r)dm (r ,, t (T). 

JT(a,ft) K J 

Theorem 4.1. For a £ iRU (0,n) and £ e Z/2Z i/ie map *(ct, fc) is if- 
equivariant between the representations 



f p°^ +1) dm^(T)MH k 

JT(<t,u) 



/k r\n>n — m\ . G 

/T( CT ,,x) 



and constructs the 7i k (M. n m ) -isotypic component in p^ £ \jj. The following 
Plancherel formula holds: 

ll*(°"i*)(/®^)lli»(R»,|(x,»)|-««da:d«) 

ll/(-' r )Hi»(H« 4|»|-B-«te) dm -^( T ) ' 

T(ct,/j) 



20 



Proof. We have already seen that ^(cr, k) gives a unitary isomorphism so that 
the Plancherel formula above holds. Further, by Proposition ^. li the map ^(cr, k) 
intertwines the actions of MhANh on smooth vectors and hence on the Hilbert 
spaces. Since H is generated by MgAN h and Ng it remains to prove the 
intertwining property for Nu- For this we use the Lie algebra action. 

Lemma 4.2. Let L be a connected Lie group with Lie algebra I and let (pi,Hi) 
and (p 2 ,'H 2 ) be unitary representations of L. Suppose that a continuous linear 
map if : T-li —> Ji 2 is given and there exist subspaces V\ C Hi and V 2 C H 2 such 
that 

(i) Vi is dense in Hi for i = 1, 2, 

(ii) Vi is contained in the space of analytic vectors Hf for i = 1,2, 
(Hi) Vi is dpi-stable for i = 1,2, 

(iv) ((p(dpi(X)vi)\v 2 )u 2 = -(^( u i)|dp2p0w2)« 2 for vi G V\, v 2 G V 2 and 

x el. 

Then tp is L-equivariant. 

Proof. For v\ G Vy and v 2 € V 2 we put 

fv u v 2 (g) ■= (<p(pi(g)vi)\v2)n 2 , 9 6 L, 

h vi,v 2 (g) : = (P2(g)(p(vi)\v 2 )n 2 = ( ( p(vi)\p2(g~ 1 )v 2 )n 2 , g e L, 

which are analytic functions on L by (ii). For a smooth function / on L and 
le [we define derivatives by 

iw/iw - ii ^T^ < £ ™<-'> = fa '"""?"''" - 

We have R{X)f(e) = -L{X)f{e) for the identity element e G L and R(X) 
commutes with L(X') for any X, X' G I. Hence 

R{X 1 )R{X 2 ) ■ ■ ■ R{X k )f(e) = - L(Xi)R(X 2 ) ■ ■ ■ R(X k )f(e) 

= -R(X 2 )---R(X k )L(X 1 )f(e) 



= {-lfL{X k )-..L{X 2 )L{X 1 )f{e) 

for X\ , . . . , X k e [. Then (iv) implies 

R(Xi) ■ ■ ■ R(X k )f VuV2 (e) = /dpi(Xi)-d P i(x fc )t;i,ti 2 (e) 

= {-t) k h Vl Ap 2 (x k )---a P2 (X 1 )v 2 (e) 
= (-l) k L{X k )---L(X 1 )h VuV2 (e) 
= R(X 1 )---R(X k )h VuV2 (e). 

Since f Vl ,v 2 and h VltV2 arc analytic functions, they coincide. Therefore <p{pi(g)vi) 
p 2 (g)(p(vi) for vi G V\ and hence tp(p\(g)v) — p 2 (g)ip(v) for any v G Hi by 
(i). □ 
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We apply the lemma to the map <p = ^(u, k) : Hi — > H2 where 



Hi := ( / i 2 (M m ,i|a;|" RoT dx)dm CTM (T) MH k {W l - m ), 



H 2 :=L 2 (W\\(x,y)\- Rc °dxdy). 
So let pi and P2 be the restrictions of 

^+r +1) d™^(T)W and p% e 

lT(<r,n) ) 

to L = Nh, respectively. We regard an element 

fe / L 2 (R m ,±\x\- ReT dx)dm a!fl (T) 

as a function f(x, r) on (R m \ {0}) x T(a, p) and let Vi be the space consisting 
of linear combinations of the functions on (R m \ {0}) x T(a,fx) x R"~ m of the 
form 

(dp^ m+1 \X)i;0(hrn + l) ){x) ^ y)x{T)j 

where X G W(fj), is the spherical vector of P^z+k as defined in 

JD2), </> G W fe (M"- m ) and x G C c {T(a,p)), i.e. x is a continuous function on 
T(cr, ju) with compact support. Further let V2 be the space of all A-finite vectors 
in L 2 (R™, \(x,y)\~ RctT dxdy). We now check conditions (i)-(iv): 

(i) V\ is dense in Hi since C c (T(a, p)) is dense in L 2 (T(a, p), dm CTjM ) and 
the space of (A' n 0(1, m + l))-finite vectors for m is generated by 
ipr m (x) and dense in L 2 (R m , |x| _Ror dx). The space V2 is dense in 
H2 since it is the space of JsT-finite vectors for pa,e' n ■ 

(ii) if-finite vectors are analytic vectors for G and in particular for Njj C G, 
hence V2 C . The inclusion V\ C "H" follows from the lemma below. 

(iii) It is clear that V2 is dp2-stable since the space of if-finite vectors is 
dpa -stable. That V± is dpi-stable follows from the definition of 

Lemma 4.3. Let 

(p'i,K) :=( T ^Sr +1) dm CT>M (r), f & L 2 (R"\ \\x\~ R ^ dx) dm, i(1 (r) ] 

A function f(x,r) on (R m \ {0}) x T(a,/i) 0/ i/ie /orra 

f(x,r) := (d^( 1 ' m + 1 )(X)^( 1 ' m + 1 ))(x) X (r) 
/or X G and x £ C c (T(cr, /i)) is an analytic vector of p^. 
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Proof. Let x — Xc + Xd be the decomposition into continuous part and discrete 

part so that X c € C c (iR+) and X d G C c (T(a, ( u)n(0, to)). Since (dp? (1,m+1) (JQ^V (1,m+: 

is an analytic vector of A^e+fc 

the discrete part (dp? (1 ' m+1) (X)^ (1 ' m+1) )(x) Xd (r) 
is an analytic vector of p^. Therefore we may and do assume x G C c (zIR+). It 
is enough to prove that for any go G 0(1, m + 1) there exists a neighborhood 
G U C So(l, m + 1) such that 



(IN 



N 1 

pi(expF)pi(ff )/(a;,T) - ^ -dpi(F)Vi(.9o)/(2;, r) 



-> 



as iV for y G U. Consider the Euclidean Fourier transform J^m with 
respect to the variable x (see (U-3[l ) which gives a unitary equivalence between 



I / 0(l,m+l) j / \ i / Ofl.ro+l) j 

Pi = / Pr.e+k 4m*A T ) and ttj := / ir y e+k dm a<fl 



(r). 



Put 



ft(x,r) := J- Rm (dp? (1 ^ i+1) (X)^° (1 ' m+1) )(x) then 



a N 



lK + 



irx (exp Y)m {g )h(x, r) 



W 1 



| X (r)| 2 dm^(r). 



As in Section 11.21 the function h(x,r) corresponds to a function h(g,r) on 
0(1, to + 1) x iR + satisfying h(gman 7 T) = t; E+ k(m)~ 1 a~ T ~ p h(g 7 T) for m G 
0(l,m+l)nM, a € A and n G iV#. Consequently, a at is given as 



!+ \yO(l)xO(m+l) 

AT 



0(l,m+l) 
r r.e+fc 



(g )ft-(cxp(-F)fc,r) 



£ ^(W) (F) ( <^r +1) Cffo)ft(*, r) 2 dfc |x(r)| 2 dm^(r) 



up to a constant factor, where dk is the Haar measure on O(l) x 0(m + 1). 
Since 7r^^™ +1 ^ (go)^ is analytic on 0(1, m + 1) x the sequence 



1=0 



converges uniformly to K®^f£ (go)h(exp(— Y)k, r) on the compact set (fc, t) G 
(0(1) x 0(m + 1)) x suppx, which proves ajv — > 0. □ 

To verify the intertwining condition (iv) we first prove the intertwining prop- 
erty for each single space L 2 (M™ , | x | ~ Ro T dx) for fixed r by embedding it into the 
C-antilinear algebraic dual of the Harish- Chandra module L 2 (M. n , \ (x, y)\~ dx dy)K 
of If -finite vectors. For r G T(a,(i) and X G U(t)) let 

f r ,x(x) := (dp^ 1 ^+ 1 )(X)^ (1 ^ +1) )(x), xeR m \ {0}. 
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Proposition 4.4. Let X £ W(fj), <p G H k (E. n ~ m ) and g € L 2 (R", \(x,y)\- Rcc7 dx dy) K ■ 
(i) For every r £ T(cr, /Lt) the integral 

[ \x\ Z ^F(^,r)f T:X (x)cl ) (y)gix~y^\(x,y)\- Rca dxdy 

Ji" 

converges absolutely and defines a continuous function in r. 
(ii) For every t £ T(cr, /j) and j = 1, . . . , m we have 

[ \x\^F(^,r)(B^ T f T , x )(x)cP(y)gJx-^\(x,y)\- R -dxdy 

JR™ 

= f \x\^F(^ 7 T)f T , x (x)<P(y)(B^g)(x,y)\(x,y)\- Rc °dxdy. 

JR™ 

(4.2) 

Proof. We first note that by (|1.13l) the function f T .x{x) is a linear combination 
of functions of the form 

f{x)=K. i+a {\x\)\x\ 2a p{x) 

for a £ No and p G C[x] with coefficients depending smoothly on r. Therefore 
we may replace f T .x{x) by one of these functions f(x). For the same reason we 
may assume that 

g(x,y) = K„^ +b (\(x,y)\)\(x,y)\ 2b q(x,y) 
for some b £ No and q £ C[x, y]. 

(i) By (|B.1[) and (|B.2[) there exists a continuous function Ci(r) > on T(a, /j,) 
and Ni > such that 

|X_x +a (t)t 2a | ^CiMa + ^e"*, i>0. 

For the hypergeometric function we have by (|B.5|) and (|B.8|) (checking the 
cases r € zK + and r <E (Re er — /i — 4No) D R+ separately) 

1^(4, r)| < C a (r)(l+t) Sa£= 5 sz=it J t > 0, 

for some continuous function C2(t) > on T(a,fi). We estimate 

|«(z,v)|<C6(l + |(*,i/)|)^ 

Further, for the -KT-Bessel function of parameter -| +& we find by (|B.1|) 
and |B~2|) that 

|iL f +6 (t)t 2b | < C 6 t~\l + t) Ni e-\ t > 0, 
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for some arbitrarily small S > (covering the possible log-term for a = 
b = 0) and N4 > 0. Hence we obtain 

Rc g-Rcr-n 

, , , , , Rc cr — Rc t ■ — /.i / |_,|2 \ 4 

< CiMCaCTjCaCkCBCslxl 3 (l + g*J 

(1 + |a:|) JVl+JVa e-l !, 'l|(a: I tf)| fc -'(l + K^^D^+^e"!^! 
<C(r)l( a!> y) r'" n t m " IUr - J (l + l( a!l y)l) jy e-l(».i>)l 

with C(r) = Ci(T)C2(r)C3C4C 5 C 6 and AT = iVi + N 2 + N 3 + N 4 . This is 
integrable on M. n with respect to the measure \(x,y)\~ Rc(T if and only if 

— Re a — n + in — Re t 



Since 5 can be chosen arbitrarily small this is equivalent to 

n — Re a + m — Rc r > 0. 

But since Re a < n and Re r < m this inequality holds true for all r S 
T(cr, /z) and therefore the integral converges absolutely. Moreover, we even 
have n— Re a+m— Rc r > n— Re tr > for all r and hence the convergence 
is uniformly in r varying in a compact subset of T(a, fi), which finishes 
the proof of (i). 

(ii) First recall from Proposition 12.11 that 

\x\^F{^,r){B^f){x)<j>{y) 

= B n ' 
3 

Therefore we have to show that 

/ B^' ,7 $(x > v).^iO.|( a:i i/)|- Re<r dxdi/ 

= / t>(x,y) ■ B^g( Xl y) ■ \(x,y)\- Rc ° dxdy, (4.3) 
ii" 

where we abbreviate 

The operator B"' a is formally self-adjoint with respect to |(x, y)\~ Rccr 
since dp^(Nj) = — i£>J ,cr is, as part of the Lie algebra action, formally 
skew-adjoint on (R" \ {0}) C L 2 (R n , \(x, y)\~ Rca dx dy)°° . Therefore 
it remains to show that we can integrate by parts without leaving any 
boundary terms. Fix j G {1, . . . , m} and consider the domain 

n jt£ := {(x, y) € R" : \ Xj \ > e} C R" 



\x\^F(H,,r)f(x)cf>(y) 
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for e > 0. Clearly E" \ U e >o 1S °^ measure zer0 an d hence (|4.3[) is 
equivalent to 



Urn / B^^{x,y)-g{x,y)-\(x,y)\-^dxdy 

= lim / $(x, y) ■ B]'"g{x, y) ■ \ (x, y) |" ReCT dx dy. (4.4) 

On Qj iS both |a;| and |(a;,y)| are bounded from below by e. Hence, by 
(|B.3j) and (|B.6[) . all factors in the integrand 



\x\-^F{^, T)f{x)<j>{y)g{x, y)\(x, y)\ 

can be arbitrarily often differentiated in x and y and the result is a smooth 
function on f2j, e . Since further the hypergeomctric function grows at most 
polynomially and the if-Bessel functions decay exponentially near oo, all 
such differentiated terms decay exponentially as |(x,y)| — > oo and are 
hence integrable on f2j. e . Therefore we can arbitrarily integrate by parts 
and all intermediate integrals exist. It remains to show that for e — > 
all boundary terms that occur while integrating by parts vanish. By the 
asymptotic behaviour of the i^-Bessel functions at oo the boundary terms 
at oo always vanish. Hence, by the choice of fij 6 j the only boundary terms 
that occur are for derivatives in Xj at Xj = ±e. Therefore we only need 
to consider the parts 

d 2 d , d 

x j^T2' n — and E ~5 — 
ox j axj oxj 

of . We treat these three parts separately. Here we start with the 
right hand side of (|4.4[) and then integrate by parts once or twice. 

(a) The boundary terms that occur when integrating by parts are 

OXj 

(up to multiplication with a constant) of the form 

Rc a 



^(x',£,y)g{x',£,y) \(x',£,y)\~ 

-${x',-e,y)g(x',-£,y) |(a;', -e,y)r Rc<T ) dx' dy 

where we write x = (x 1 , Xj) with x' = {x\, x m ) € R m_1 . 

The integrand obviously converges pointwise almost everywhere to 
as e — > and it suffices to find an integrable function independent 
of £ dominating the integrand to apply the Dominated Convergence 
Theorem. For this note that in both 3>(x,y) and g(x,y) the only 
terms dependent on the sign of the polynomials p(x) and 

q(x, y), respectively. Using the same estimates as in the proof of part 



2G 



(i) we find that 



^(x',e,y)g(x',e,y)\(x , ,e,y)\-^ 

-$(x', -e, y)g(x', -e, y)\(x', -e, y)\ 
< C\(x>,e,y)\ ~ R S m - R ° T - s (l + \{x>,e,y)\re-^'^)\ 



for some N > and an arbitrarily small 8 > 0. Now note that 
p(x' , s)q(x' , e, y) — p(x', —s)q(x', —e, y) is an odd polynomial in e and 
hence of the form e ■ r(x',e,y). For the extra e from this obser- 
vation we use the estimate |e| < \(x',e, y)\. We further estimate 
\r(x',e,y)\ < C"(l + \(x', e, y)\) N ' for some C',N' > and find (as- 
suming e < 1) 



< CC'\(x\e,y)\^ RRJL ^ m ^ ! ^ +1 ^ 5 {l + \(x', l,y)\) N+N 'e~^ x '' v)l . 



Now suppose the exponent — - " 2 +m T + 1 is < 0. Then we can 



<CCl(^,y) r Rc ^ + "" RcT +^(l + |(xM,;/)|)^ e H^^)l 



which is independent of e G (0, 1) and intcgrablc on K n_1 for small 
5 > since Re a < n and Rc r < m. If the exponent -R eg -"+ m -R eT -|- 
1 — <5 is positive the estimate e < 1 also yields a dominant intcgrablc 
function independent of e. Therefore, in both cases we can apply 
the Dominated Convergence Theorem and obtain that as e — > the 
boundary terms vanish. 

(b) XjS^j. Integrating by part once gives (up to multiplication by a 

j 

constant) the boundary terms 



-<!>(x',-e,y) l Xj -Z-( x ,y)) |(z', -e, y)|~ RcCT ] dx' dy. (4.5) 



and use the product rule to find Xj-g^-(x,y). The first term is by 



p(x', e)q{x', s, y) - p(x', -e)q(x', -e, y) 



estimate 




We have 



g(x,y) = K-^ +b (\(x 7 y)\)\(x,y)\ 2b q(x,y) 




% +b+1 (\(x,y)\)\(x,y)\ 2b q(x,y) 



K_ §+b+1 (\(x,y)\)\(x,y)\ 2{b+1) <i(x,y) 
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and putting xj = ±e gives 



;K-, +b+1 (\(x',e, y)\)\(x', e, y)\ 2 ^g{x\ ±e, y). 



2\(x',e,y)\*- 

Again e 2 can be estimated by \(x',e,y)\ 2 and we find that 

r A^ f+b+1 (|(x', £ , 2/ )|)|(x', £;2/ )| 2 ( b+1 )and K_ %+b (\(x,y)\)\(x,y)\ 



2b 



2|(^',£,y)| : 

satisfy the same estimates (see part (i)). The same argument applies 
to the other two terms in the product rule. Therefore the same 
argument as in (a) yields the vanishing of the boundary terms (|4.5[) . 
Similar arguments yield the vanishing of the boundary terms that 
occur when integrating by parts for the second time. For this note 
that the formal adjoint of g§- on L 2 {K n , \(x, y)\~ Rca dxdy) is —£ r + 
(Reg) 1(^)1 a ■ Both summands are treated separately as above. 

(c) E-£-. We have 

d d 2 x - d d \ - d d 

dxj dx 2 ^ k dxk dxj ^ ^ k dyu dxj 

The first term was already treated in part (b). For the other two 
terms note that we can first integrate by parts the derivatives with 
respect to Xk (k ^ j) and yk without any boundary terms occurring. 
Secondly, integration by parts of the derivative with respect to Xj is 
dealt with as in part (b). This finishes the proof. □ 

Remark 4.5. It is necessary in the proof of Proposition 14.41 (ii) to restrict 
integration to the domain Oj, E . This is because the operator B™ ,a is of second 
order and we have to integrate by parts twice. The intermediate result, i.e. after 
integrating by parts once, may not be integrable on M" and hence we need to 
restrict to a subdomain on which these intermediate results are integrable. The 
same problem occurs when one considers the two summands XjA and ~(2E — 
a + n) separately. Here the integral over R™ for each of the two summands 
may not converge while the integral for the sum S™' 17 does by Proposition l4.4l (i) 
converge. 

Remark 4.6. Part (i) of Proposition l4~4l constructs an embedding of L 2 (M. m . \x\~ ReT dx)KnO(i,m+i)l 
n k (M. n ~ m ) into the C-antilinear algebraic dual of L 2 (W l , \(x, y)\~ Rca dx dy) K 
for every r £ T(a, /i). By part (ii) this embedding is f)-equi variant. 

Let us now continue the proof of Theorem 14.11 by showing property (iv) in 
Lemma |4~21 Let V\ £ V\ and £ V<x. Suppose that 

vi[x,T,y) = f r ,x(x)(/)(y)x(r) and v 2 (x, y) = g(x, y) 



with A £ W(fj), x 6 C c (T(a,fi)), (j) £ n k (R n ~ m ), and g £ L 2 (R n 7 \(x, y)\~ Rca dx dy) 



K ■ 
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We have 

(^(d/9l(iYj>i)|u2)H 2 



= -i / / \x\*F{^,T){B?> r f T>x ){x)cj>{y)g{x,y)\{x,y)\- 

X(r) dm^^r) da; dy 

= -i/ / \x\^F(^,T)(Bj t - T f T , x )(x)^y)W^)\(^y)\- Roa 

X(r) da; dy dm^^r), 

where we were able to change the order of integration, because by Proposition 
14.41 (i) the inner integral in the last line converges absolutely and is continuous 
in t and the integration is only over the compact subset supp \ 5= T{o~, /i). Now, 
by Proposition 14.41 (ii) we find 

= ~i[ I \x\^F(^,r)f T , x (x)cP(y)BYgJx~y~)\(x, y )r R - 

x(r) dx dy dm CT)Jli (r) 
\x\^F(^ : T)f T , x (x)<P(y)B^g(x : y)\(x,y)\- Rc ° 

X(r) dm CTiAI (r) da; dy 

= (<p(vi)\dp2(Nj)v 2 ) n2 , 

again using Proposition 14.41 (i) to change the order of integration. This finally 
shows property (iv) of Lemma 14.21 and we obtain that (p = ty(o~,k) intertwines 
the group action of Nh and hence of H. Thus the proof of Theorem 14.11 is 
complete. □ 

We obtain the whole spectral decomposition of p^ e \n from (|4.1I) and Theo- 
rem [O] 

Theorem 4.7. For a £ ilU (— n, n) the representation p^ e decomposes under 
the restriction to H = 0(1, m + 1) x 0(n — m) as 



fe=0 V 7 * 



0(l,m+l) , 
Pr,e+k dr 



jgZn[o, |R " | -; + "" 2 " ) / 

5 Intertwining operators in the non-compact pic- 
ture 

In Proposition ^ . 1 1 we explicitly found an intertwining operator C°°(R m \ {0}) IEI 
H k {M. n ~ m ) -> C°°(R" \{x = 0}). In the Fourier transformed picture this 
operator is given by 

A(a, r)(f ® d>)(x, y) = \x\^ 2 F, f ; - j|D /(*)*(*), 
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where again fj, = 2k + n — m. In Proposition 14.41 we even showed that for 
fixed a £ iR U (— n, n), k 6 No and r € T(a, 2k — n + m) the operator A(a, r) 
is intertwining between the Harish-Chandra module of p°^f£ ^ % k (W 1 ^ 111 ) 

and the C-antilinear algebraic dual of the Harish-Chandra module of pa,e' n - 
We now find a formal expression for this intertwiner in the non-compact picture. 

Consider the following diagram 

C£°(M m \ {0}) <g> H k (W 1 -" 1 ) 

.y R mlgiid 

" \ {o}) ®n k (w i - m ) — 



A(a,r) 



■S'(R") 

Fun 



■S'(R"). 



We extend the operator A(<r, r) for all <r, r e C and determine the operator 
I(cr, t) for Re a < Re r < 0. We have 



J- R ^(a,r)(/®0)(^ry) 

(27T)" 



g-ix-e-is/^ia-i 



We first calculate the integral over y 6 
integral formula (|B.9|) we find 



(27T)" 



. /X-C7 + T fJ,-a-T /J |?/| 2 \ , n , 

j*i ( 4 > 1 ; 2 ;_ ^ J /(a;)0(y)dydx. 

Using Appendix IB. 41 and the 



f -tyn j? f^-CT + T H-a-T fJL \y\ 2 \ 



rVWM - ** 2 / J^(\v\shF : 
Jo 



4 '2' |x| 

jU — (7 + T /! — (7 — T /i 



i2 ds 



If we let 

^(a;,r?) 
then we find that 



2^r(H) 



r(i^|±i)r(im) 

2r_ 



H-^Kx(H-hl). 



2^~T(§) 



)r(i 



N-^(N-N) 



7k»A((r, r)(/ ® , 77) = J" Rm (/ • *?))(£) ■ ^»(r?) 

= (j- Rm V(-,^)^ R ™/)(0-^)- 



Therefore we compute, using again Appendix IB. 41 and the integral formula 
(fBTTOl) (noticing that K„(x) = K- U {x)) 



2"*V fc r(§) ^ 



r(i -£±I )r( £i 4 

2 s= 5 tffi i-*r(f)r(2^i) 



r( ^ f±I)r(/ ^ £) 
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Altogether we see that J(<7, /i) is a partial convolution operator combined with 
a multiplication operator 

H<r,T)(f®m,V) = const ■ H"^^) / (|£ - £'| 2 + M 2 )^/(0 d£'. 

For m = n — 1 this operator appears in jlOj as a special case. This expression 
for /(a, r) is valid for Rcer <C Rer <C 0. It has a holomorphic extension to all 
a, r G C for / G Jr^O^R" 1 \ {0}). 

A Decomposition of principal series 

We give a short alternative proof for the decomposition of the principal series 
7t£t e , a G iR, e G Z/2Z, into irreducible P-representations. This decomposition 
turns out to be essentially equivalent to the Plancherel formula for L 2 (0(1, m + 
1)/(0(1) x 0(m + 1)),C' S ), where C' s are the line bundles over the Ricmannian 
symmetric space 0(l,m + 1)/(0(1) x 0(m + 1)) induced by the characters 
(a, g) i-> a s of 0(1) x 0(m + 1), 6 G Z/2Z. 

Consider the flag variety X = G/P. Since G/P = K/M we can identify X 
with the unit sphere S n C R n+1 . For this we define a G-action on S" by the 
formula 

ffox:= pr £ Ml 1 ^l ) xe5 „ 
P r o(5(l^)) 

where pr : R ra+2 — >• R and pr^, : R n+2 — > R n+1 denote the projections onto the 
first coordinate and the last n + 1 coordinates, respectively, and g(l,x) is the 
usual action of g on (l,x) G R x R n+1 = R"+ 2 . Then it is easy to prove the 
following: 

Lemma A.l. The operation o defines a transitive group action of G on S n . 
The stabilizer of the point e n +i = (0, ... ,0, 1) G 5™ is equal to the parabolic 
subgroup P. The maximal compact subgroup K also acts transitively on S n and 
the stabilizer subgroup of the point Xq is equal to M . 

Let us consider a slightly different embedding of O ( 1 , m + 1 ) x O (n — m) into 
G= 0(1, n + l). Let 

H' := {diag(.g, h) : g G 0(l,m + 1), h G 0(n - m)}. 

Then clearly H and H' arc conjugate and hence the branching to H is equivalent 
to the branching to H'. We shall therefore only deal with H' in this section. 

Lemma A. 2. Under the action o of the group W the sphere S n decomposes 
into the two orbits 

:=H'o ei = {(x\0):x'eS m }, 

01 := H' o e„+i = {(x',x") G S n : x' G R m+ \x" G R"- Tn ,x" ^ 0}. 
The orbit 0\ is open and dense in S n . The isotropy group of e n+ \ in H' is 

S = {(a,g,h,a) : a G 0(1), g G 0(m + l),/i G 0(n - m — 1)}. 
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Now consider the realization of 7r CTi£ in the compact picture, i.e. on L 2 (G/P, C a>s ), 
where denotes the line bundle over G/P associated to the character man <— > 
£ >e {m)a a+p of P. Since the orbit 0\ C G/P is open and dense we have 

L 2 (G/P,L a , e ) ^ L^CL^loJ. 

Now the stabilizer S of eP £ G/P in H is contained in P and hence the 
restriction of the line bundle C a , e to 0\ = H' / S is induced by the restriction of 
the corresponding character of P to S which is simply £ e \s- Therefore we find 

L 2 {G/P,C <x , s )^L 2 {0 1 ,C e ), 

where C e is the line bundle over 0\ = H'/S induced by the character £ e |s. 
Using the decomposition of L 2 (S n ~ m ~ 1 ) into spherical harmonics we find 

oo 

L 2 (G 1; Ce) <* ]T® L 2 (0(l,m + 1)/(0(1) x 0(m + 1)), C' £+k ) E H fe (M"- m ) 

k=0 

as ^'-representations, where for S £ (Z/2Z) we denote by C' s the line bundle 
over the symmetric space 0(1, m+l)/(0(l) x 0(m+l)) induced by the character 
(a, g) i — y a 5 of 0(1) x 0(m + 1). Together we obtain 

oo 

<e\ H = E L 2 {0(\,m + 1)/(0(1) x 0(m + K 

fc=0 

and hence the decomposition of 7r^ e |# into irreducible ii-representations is 
equivalent to the decomposition of L 2 (0(1, m + 1)/(0(1) x 0(m + 1)), into 
irreducible 0(l,m + l)-representations, 5 <G Z/2Z. Since 0(l,m + 1)/(0(1) x 
0(m + 1)) is a Riemannian symmetric space of rank one the decomposition of 
L 2 (0(l,m + 1)/(0(1) x 0(m + 1)),C' S ) is well-known and given by 

L 2 (0(l,m + l)/(0(l)xO(m + l)),4)= T 7r^ 1 ' ra+1) dr, 

the unitary isomorphism established by the spherical Fourier transform. This 
proves Theorem 14. 71 for the special case a € iR. 

B Special functions 

For the sake of completeness we collect here the necessary formulas for certain 
special functions needed in this paper. 

B.l The K-Bessel function 

We renormalize the classical X-Bessel function K a (z) by 

K a (z) := ^)~ a K a (z). 

Then K a (z) solves the differential equation 

d 2 u 2a + 1 du 
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It has the following asymptotic behaviour as x — > (see jTSJ Chapters III & 
VII]): 

^(f)" 2a + o(a;- 2Q ), forRea>0, 
K a (x)= ^-log(f)+o(log(|)), for Re a = 0, (B.l) 
l: ^^o(l), for Re a < 0. 



2 

Further, as x — > oo we have 

*.M-£(5)~''~( 1 + °(i))- (B - 2 » 

For the derivative of K a (z) the following identity holds (see [T5J equation 
111.71 (6)]): 

^Mz) = -~#a+i(*). (B.3) 

B.2 The Gaufi hypergeometric function 

Consider the classical Gaufi hypergeometric function 

0)n(&)n „ 



2 Fi(a,6;c;z) = ^ 



n!(c) r 



where (a)„ = a(a + 1) ■ ■ ■ (a + n — 1) denotes the Pochhammcr symbol. The 
function 2Fi(a,b] c; z) is holomorphic in z for z ^ [l,oo) and meromorphic in 
the parameters a, fe, c 6 C. It solves the differential equation 

* d 2 it . „ du j . 

(1 - z)z— + (c-(a + b+ l)z) — - abu = 0. (B.4) 
dz z dz 

The following formula allows to study the asymptotic behaviour of the Gaufi 
hypergeometric function near x = oo (see [3J equation 9.132 (2)]): 

2 F 1 (a, b; c; z) = E^-ZiHM (_ g )-« aFl ( , a — c + 1; a — 6 + 1; |) (B.5) 
1 (ojl (c — a) 

Both summands on the right hand side of ()B.5|) are gcncrically linear indepen- 
dent solutions of (|B.4[) . Their Wronskian is given by 

W{z- a 2 F 1 {a, a-c+l-a-b+l; -i), «- 6 2 Fi(6, 6 - c + 1; & - a + 1; -I)) 

= (o-6)(l + z) c - a - b - l z~ c . 

The following simple formula for the derivative of the hypergeometric function 
holds: 

^- 2 Fi(a,b;c;z) = — 2 F 1 (a + 1, b + 1; c+ 1; z). (B.6) 
dz c 
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We recall Kummer's transformation formula (see [51 equation 9.131 (1)]): 

2 F 1 (a, b; c; z) = (1 - z) c - 1 - b 2 F l (c - a, c - 6; c; 2). (B.7) 



For a G —No the hypcrgcometric function 2-P\(a, &; c; z) degenerates to a poly- 
nomial which can be exp 
equation 8.962 (1)]): 



nomial which can be expressed in terms of the Jacobi polynomials P„ ' (z) (see 



2 F 1 (-n,b;c;z) = 7 ^Pi c - 1 - b - c - n \l-2z), n G N , (B.8) 

where 

fc 



(-n)fe(a + 6 + n + l) k (a + k + l) n _ fc /l - 2 



ri! ' — ' fc! 
B.3 Integral formulas 

We consider the J-Bessel function J u (z) and the X-Bessel function K v (z). For 
the J-Bessel function and the hypergeometric function the following integral 
formula holds for y > 0, Rc A > and — 1 < Rc^ < 2 max(Rc a, Re f3) — | (see 
equation 7.542 (10)]) 



2 F 1 {a, + -A 1 )J„(ij/)i" + dx 

2 ,-^ +2l> + 1) 

- A^r( a )r(/?) y ^"HaJ- (R9) 

For the J-Bcsscl function and the A"-Bcssel function we have the following inte- 
gral formula for Re fj, > | Re v\ — 1 and Re b > | Im a\ (see equation 6.576 (7)]) 

jf ^ +1 ^M^(M d, = 2^W^^L. (B.10) 
B.4 Fourier and Hankel transform 

Let J-"r" denote the Euclidean Fourier transform on R n as defined in (|1.3|) . Let 

G N and G "H fc (]R™). For / G L 2 (R+, r^ 2 *" 1 dr) denote by /<g>0 G L 2 (R") 
the function 

(/®0(a;) :=/(H)0(x), x G R™. 

Then by pS Chapter IV, Theorem 3.10] 

Tv>{j ®4>) = i~ k CH n+ 2k-i f) <g> 0, 
where H,, is the modified Hankel transform of parameter v > — | 

/>oo 

H v f(r)=r-» Mrs)f( s )s» +1 d s , 
Jo 

which is a unitary isomorphism (up to a scalar multiple) on L 2 (R_|_, r 2u+1 dr). 
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